Skip to content

zfs(openzfs): cap the default ARC at half of RAM for app coexistence - #1487

Draft
gburd wants to merge 2 commits into
cloudius-systems:masterfrom
gburd:pr/zfs-arc-halfram-cap
Draft

zfs(openzfs): cap the default ARC at half of RAM for app coexistence#1487
gburd wants to merge 2 commits into
cloudius-systems:masterfrom
gburd:pr/zfs-arc-halfram-cap

Conversation

@gburd

@gburd gburd commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

Draft / follow-on to #1423 and #1478.

On OSv the ARC and the application share one physical address space. arc_default_max() for a guest with >= 1 GiB of RAM returns MAX(allmem*5/8, allmem-1GiB), which on a large guest is nearly all of RAM (about 31 GiB of a 32 GiB guest). A memory-hungry application such as PostgreSQL (multi-GiB shared_buffers plus many concurrent backends) then has almost no headroom, and a checkpoint burst stacking dirty ARC buffers on top of the backends' working set drives the guest OOM and aborts it. This showed up as an OOM at high concurrency on a 32 GiB guest.

Since ZFS_MODULE_PARAM is a no-op on OSv (zfs_arc_max cannot be set at boot), this caps the computed default at half of RAM for guests with more than 2 GiB, so the application always keeps at least half the guest's memory. Small guests are unchanged.

Draft because the proper long-term fix is to wire zfs_arc_max to the boot cmdline so this becomes an overridable default rather than a hard cap; kept as a draft PR so the tuning is not lost while the maintainer reviews the ZFS follow-on stack. Edits the tracked modules/open_zfs/osv/module/os/osv/zfs/arc_os.c directly (the OSv platform layer is now source files on master, not a numbered patch).

On OSv the ARC and the application share one physical address space. The
existing arc_default_max() for a guest with >= 1 GiB of RAM returns
MAX(allmem*5/8, allmem-1GiB), which on a large guest is nearly all of RAM
(for example ~31 GiB of a 32 GiB guest). A memory-hungry application such as
PostgreSQL, with several GiB of shared_buffers plus many concurrent backends,
then has almost no headroom, and a burst (a large checkpoint stacking dirty
ARC buffers on top of the backends working set) drives the guest out of
memory and aborts it.

Since OSv ZFS_MODULE_PARAM is a no-op (zfs_arc_max cannot be set at boot),
cap the computed default at half of RAM for guests with more than 2 GiB, so
the application always keeps at least half the guest memory. Small guests are
unchanged (their fractions already leave room). A future zfs_arc_max wired to
the boot cmdline could override this.

Follow-on to the OpenZFS keystone (cloudius-systems#1423) and the runtime data pool (cloudius-systems#1478).

Signed-off-by: Greg Burd <greg@burd.me>
…r OOM

Companion to the half-RAM ARC cap: even with the ARC capped, at high
concurrency (many PostgreSQL backends plus a large checkpoint and autovacuum
worker forks) the OSv memory reclaimer can abort with out-of-memory while the
ARC still holds evictable pages. OSv's reclaimer calls the ARC shrinker and, if
that pass frees nothing, OOMs. The hard-mode shrinker only called
arc_reduce_target_size_noshrink(), which lowers the target and wakes the async
evict thread but returns before any page is freed, so under a burst the
reclaimer saw zero freed bytes and aborted even though eviction was in flight.

Add arc_reduce_target_size_shrink_sync(): it lowers the target the same way and
then waits in arc_wait_for_eviction() until the requested amount is actually
evicted (lax so it returns early once the ARC stops overflowing). The hard-mode
shrinker now calls it, so the reclaimer gets real freed memory before deciding
it cannot reclaim; soft mode stays asynchronous. The eviction waiter is already
fork-COW coherent and the reclaimer thread may block.

Measured on a large-working-set PostgreSQL-on-ZFS run: a virtual-user level
that previously aborted with 'could not reclaim any further' now completes,
riding through a large mid-run checkpoint, with no reclaimer OOM, no deadlock,
and no throughput regression at the level below.

NOTE: the arc.c hunk is applied via the vendored-edits patch; if a hunk offset
drifts, apply with git apply --recount.
@gburd

gburd commented Aug 31, 2026

Copy link
Copy Markdown
Contributor Author

Added a second commit: a synchronous hard-mode ARC shrinker, the companion to the half-RAM cap. Even with the ARC capped, at high concurrency (many backends + a large checkpoint + autovacuum worker forks) OSv's reclaimer could abort out-of-memory while the ARC still held evictable pages, because the hard-mode shrinker only lowered the target and woke the async evict thread, returning before any page was freed. The new arc_reduce_target_size_shrink_sync() lowers the target and then waits in arc_wait_for_eviction() until the requested memory is actually evicted, so the reclaimer gets real freed memory before deciding it cannot reclaim; soft mode stays async.

Measured: a virtual-user level that previously aborted with 'could not reclaim any further' now completes, riding through a large mid-run checkpoint, with no reclaimer OOM, no deadlock, and no throughput regression at the level below. Kept as part of this draft since both commits are ARC memory-pressure coexistence fixes for the same app-shares-RAM situation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant